Lecture 2

Solution One quick way to get the answer (but not the proof) is to test the question for very small numbers. Suppose n = k = 2 and the bin contains one red and one blue ball. Then the possible outcomes of the first experiment are RR, RB, BR, and BB – all with equal probabilities. Hence the entropy of the first experiment is 2 bits. The possible outcomes of the second experiment are RB and BR, and hence the entropy is only 1 bit. Thus we should try to prove that the first experiment has the higher entropy. Denote the random variables representing the colors of the balls in the first experiment by X1, . . . , Xk and in the second experiment by Y1, . . . , Yk. The first experiment is run with replacements, and hence all the Xi’s are independent and identically distributed. Thus we have: